The influence of clinical coaching teams on quality of entrustable professional activity assessments

Abstract Background Coaching is an important component of workplace‐based assessment in competency‐based medical education. Longitudinal coaching relationships have been proposed to enhance the trainee–supervisor relationship and promote high‐quality assessment. Objective The objective of this study was to determine the influence of longitudinal coaching relationships on the quality of entrustable professional activity (EPA) assessments. Methods EPAs (n = 174) completed by emergency medicine (EM) supervisors between July 2020 and June 2021 were extracted and divided into two groups; one group consisted of EPAs completed by supervisors when a longitudinal coaching relationship existed (n = 87) and the other group consisted of EPAs completed by the same supervisors when no coaching relationship existed (n = 87). Three physicians were recruited to rate the EPAs using the Quality of Assessment and Learning (QuAL) score, a previously published measure of EPA quality. An analysis of variance was performed to compare mean QuAL scores between the groups. Linear regression analysis was conducted to examine the relationship between trainee performance (EPA rating) and EPA assessment quality (QuAL score). Results All raters completed the survey. The mean ± SD QuAL score in the coaching relationship group (3.63 ± 0.91) was higher than the no coaching relationship group (3.51 ± 1.10) but the difference was not statistically significant (p = 0.40). Supervisor was a significant predictor of QuAL score (p = 0.012) and supervisor alone accounted for 26% of the variability in QuAL scores (R2 = 0.26). There was no significant relationship between trainee performance and EPA assessment quality. Conclusions The presence of a longitudinal coaching relationship did not influence the quality of EPA assessments.

Objective: The objective of this study was to determine the influence of longitudinal coaching relationships on the quality of entrustable professional activity (EPA) assessments.
Methods: EPAs (n = 174) completed by emergency medicine (EM) supervisors between July 2020 and June 2021 were extracted and divided into two groups; one group consisted of EPAs completed by supervisors when a longitudinal coaching relationship existed (n = 87) and the other group consisted of EPAs completed by the same supervisors when no coaching relationship existed (n = 87). Three physicians were recruited to rate the EPAs using the Quality of Assessment and Learning (QuAL) score, a previously published measure of EPA quality. An analysis of variance was performed to compare mean QuAL scores between the groups. Linear regression analysis was conducted to examine the relationship between trainee performance (EPA rating) and EPA assessment quality (QuAL score).
Results: All raters completed the survey. The mean ± SD QuAL score in the coaching relationship group (3.63 ± 0.91) was higher than the no coaching relationship group (3.51 ± 1.10) but the difference was not statistically significant (p = 0.40). Supervisor was a significant predictor of QuAL score (p = 0.012) and supervisor alone accounted for 26% of the variability in QuAL scores (R 2 = 0.26). There was no significant relationship between trainee performance and EPA assessment quality.

Conclusions:
The presence of a longitudinal coaching relationship did not influence the quality of EPA assessments.

INTRODUC TI ON
The role of clinical supervisors in competency-based medical education (CBME) has evolved to include a focus on observation of trainees in the workplace. 1 These observations provide an opportunity to move beyond traditional approaches to feedback and engage in coaching. 2 Historically, feedback to trainees has been of poor quality and low reliability and often does not lead to actionable suggestions for performance improvement. [3][4][5][6][7][8] Episodic, fragmented, and limited interactions between supervisor and trainee act as barriers to meaningful assessments. 9 The unwillingness of supervisors to provide negative feedback to trainees with whom they have limited familiarity has led to the development of politeness strategies and the "failure to fail" phenomenon to smooth social functioning. 4,10,11 Longitudinal coaching in medical education has been introduced to enhance the trainee-supervisor relationship, promote continuous feedback, and improve clinical performance. 12 Increased time spent with trainees has previously been shown to help supervisors provide more specific and individualized feedback, witness performance improvement as a result of feedback exchange, and identify trainees in difficulty. [13][14][15][16][17][18][19][20] However, the impact of longitudinal coaching initiatives on the quality of entrustable professional activity (EPA) assessments has not been studied. 21 Workplace-based assessments (WBA) such as EPAs are critical for trainees to fuel growth as well as for residency program leaders to make decisions pertaining to trainee promotion and readiness for independent practice. The resource intensive nature of coaching and the reliance on EPAs for trainee progression in CBME warrants an investigation into the value of such relationships. 22 The objective of this study was to evaluate the impact of supervisor-trainee continuity in the form of longitudinal clinical coaching teams (CCT) on the quality of EPA assessments.

Setting and population
This study was conducted in the department of emergency medicine (DEM) at the University of Ottawa, Canada, in 2021. 23

Design
EPAs from trainees (n = 20) during the study period (July 2020-June 2021) were extracted, deidentified, and segregated based on completion following a CCT or non-CCT (NCCT) interaction in the emergency department. During the study period, 29 teaching faculty participated in CCTs. For each supervisor, six assessments (three CCT and three NCCT) were randomly selected. Supervisors with less than three CCT or NCCT EPAs were excluded to maintain an equal number of assessments from all supervisors.
An online survey was created to present these EPAs in a randomized order. Three EM physician raters were recruited to rate the quality of the assessment documented on the selected EPA forms using the QuAL score. A previous study has shown that the QuAL score requires three raters to reach an acceptable target reliability of 0.79. 25

Outcome measure
The QuAL score was used to objectively rate the quality of short, workplace-based comments contained within EPA assessments. 26 A QuAL score of 5 represents the highest possible score whereas a score of 0 is the lowest possible score. Validity evidence for the QuAL score has been demonstrated for short WBA such as EPAs, including high reliability and strong correlation with rater perception of utility. 27

Data analysis
A comparison of mean QuAL scores between the CTT and NCCT study groups was conducted. A factorial analysis of variance (ANOVA) was conducted with mean QuAL score as the dependent variable and presence of a CCT relationship as the independent variable. ANOVA analysis with only supervisor as an independent variable was also performed for comparison. The relationship between the trainee performance (given by the EPA entrustment scale rating) and QuAL score was examined using a linear regression analysis with EPA entrustment scale rating as the independent variable and QuAL score as the dependent variable. Separate linear regression analyses were conducted for both the CCT and the NCCT groups.

Ethics statement
This study received ethics exemption from the Ottawa Health Science Network Research Ethics Board.

RE SULTS
A total of 174 EPAs completed by 29 supervisors (CCT 87, NCCT 87) were rated using the QuAL score by all raters (n = 3). Of the EPAs rated, 33 supervisors contributed EPA assessments for 20 trainees.
The mean ± SD number of CCT shifts per resident was 15.8 ± 5.3
Supervisor alone accounted for 26% of the variability in QuAL scores (F = 1.82, p = 0.012, R 2 = 0.26). Adding in the presence of CCT to supervisor did not explain any additional variability in QuAL score compared to supervisor alone. There was no relationship between trainee performance (EPA entrustment scale rating) and EPA assessment quality (QuAL score) (p = 0.93).

DISCUSS ION
Our study did not demonstrate any significant impact of supervisortrainee continuity in the form of longitudinal CCTs on the quality of EPA assessments. The results align with those of a similar study by Cheung et al. 9 performed in a non-CBME curriculum. EPA assessments only represent the documented aspect of feedback and do not encompass the verbal feedback that occurs in the moment.
However, competence committees are not privy to verbal feedback and rely on documented EPA data to make high-stakes progression decisions.

LI M ITATI O N S
Although some studies have demonstrated the superiority of feedback obtained during longitudinal supervisor-trainee relationships, their results are derived from qualitative analysis based on the perception of assessments by supervisors and trainees. 14,15,[17][18][19][20] This is perhaps a more effective model to assess the role of continuity on resident learning rather than documented assessments. Furthermore, we do recognize that the longitudinal CCT model is unique to our institution and other programs may have implemented forms of enhanced supervisor-learner continuity differently, limiting the generalizability of our findings.
Scheduling challenges make standardization of continuity difficult. It is possible that certain randomly selected EPAs were from early interactions between trainee and supervisor. However, Telio et al. 28 demonstrated that residents judged their educational alliance by the supervisor's engagement as an educator rather than the amount of time spent together. The complexity of feedback encompasses the role of learner culture, perceptions of credibility, and trainee receptivity to feedback which may not be mitigated by increasing continuity. 29 In this study, supervisors were more significant predictors of the quality of EPA assessment than supervisor-trainee continuity.
There remains heterogeneity in the capability of individual supervisors to document high-quality assessments of residents. 30 Faculty development has been shown to modestly improve the quality of written feedback and additional faculty development related to EPA assessment documentation may be of benefit. 7,[30][31][32] Although an inverse relationship between quality of assessment and trainee performance has previously been observed in non-CBME systems, 3,9 our study failed to replicate this finding. One hypothesis is that EPAs are more task oriented compared to other forms of WBA and may lend itself to higher quality assessment regardless of trainee performance. Future studies involving a qualitative analysis of trainee and supervisor perspectives on longitudinal coaching relationships may offer a deeper understanding of the perceived benefits and challenges of coaching in medical education as well as its role in learning and assessment.

CON CLUS IONS
Increased supervisor-trainee continuity in the form of longitudinal clinical coaching teams did not demonstrate a significant improvement in the quality of entrustable professional activity assessments.

FU N D I N G I N FO R M ATI O N
This study was funded by an internal research grant from the Department of Emergency Medicine, University of Ottawa.